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ABSTRACT 


"...time study...may be defined as a procedure for 
determining the amount of time required, under certuin 
standard conditions of measurement, for tasks involving 
some human activity. “1 One of the most important phases of 
a time study is that of rating the job pace or rate of ac- 
tivity. It is here that judgment on the part of the rater 
plays a mejor role. The major objective of this thesis is 
to examine the current state of time-study rating. 

A group of seventy-two raters, nearly all active time 
study engineers, gathered at the 1950 Purdue Work Session, 
were the cross-section of the profession whose work furnishe 
ed the basis for this study. 

A series of eighteen films, three paces each of six 
Gifferent actual industrial jobs, were shown in entirely 
random order of pace and job. Each rater assigned ratings 
exactly as if the jobs were being performed in his own 
plant, according to his own company standard and company 
policies. These ratings were converted by him to a common, 
comparable, numerical basis to facilitate further analysis. 

Previous to the showing of the films, personal data 
questionnaires were individually filled out, thus providing 
the necessary information for assembling the data according 


to various parameters, 





1. M. Be Mundel, Motion and Time Study Principles and 
Practices, New York, Prentice-Hall Inc., 1950, p. l, 


(From Manuscript). 
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The data so obtained was arranged and computations 
made in such a manner as to permit of analysis as to cone 
sistency, sccuracy, and amount of variation. Also, it was 
desired to ascertain whet outstanding differences, if any, 
might be found with regard to the following parameters: 
years experience, concept of standard, initial training, 
size of rater's own plant, geographic area, and size of 
city in which plant is located. 

The ratings on each film were arrayed for the whole 
group and by each subdivision of the major parameters. (Con- 
sistency is a measure Of how the individual ratings are 
grouped about their arithmetio mean; accuracy, of how they 
are grouped about a least-squares corrected mean. 

In order to carry out the other objectives, it became 
necessary to find an additional basis for comparison be- 
tween the various parameter subdivisions, this basis to be 
capable of consolidation of much bulk into a few statistics 
that will submit to further analysis. Thus, for each ar- 
ray, limits were set up at the 45%, *744, %104, and the 
220% levels about the actual means to measure consistency, 
and similar limits were set about the corrected means to 
measure accuracy. Percentages of each array lying within 
each set of limits were then determined, becoming the 
figures for comparison. 

For consistency of rating, within *6% of the group 
mean we find an average of 38.2 percent or nearly two out 


of five, which increases to 64.9 percent, nearly 2 out 3, 
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within 410%. The results for overall group accuracy are 
somewhet lower, ranging from 1 in 3 within 25¢ to nearly 
3 out of 5 within +10% of the corrected mean. 

The average range of the ratings of pace for the 
whole group, still based on 100 as standard, is found to 
be 77 - 154, exactly a 2 to 1 spread. The averege rating 
to which such an average range applies is 115 percent of 
stenderd. 

It is felt that this analysis of “current practice” 
time-study rating shows that the present state of time- 
study rating is not up to the level required for the wide- 
spread confidence and acceptance thet is so necessary for 
its use. The need for time studies and their proven aid in 
solving many complex industrial problems have been demon- 
strated for many years. The need now is for more consistent, 
accurate, and reliable ratings of pace and their end pro- 
duct, more absolute, trustworthy, and authoritative standard 


times. 
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AN ANALYSIS OF CURRENT-PRACTICE, UNAIDED, 
TIME-STUDY RATINGS 


INTRODUCTION 


",.time study...may be defined as @ procedure for 
determining the amount of time required, under certain stan- 
dard conditions of measurement, for tasks involving some 
human activity. "2 fhe end result of a time study is the 
standard time for the job. One of the most common types of 
time study is the stop-watch time study, by far the most 
widely used aid in obtaining a stenderd time,” as defined 
by Mundel: "A stop-watch time study is used to find the 
amount of time that will be necessary to perform a unit of 
work, using a given method, under given conditions of work, 
by a worker possessing sufficient skill to perform the job 
properly, as physically fit for the job after adjustment 
to it as the average person who can be expected to be put 
on the job and working at a pace 100/130 of the maximum 
pace that can be maintained, day after day, without herm- 
ful physical effects. "© 


It is obvious that it would be mere chance to find a 





1. MM. B. Mundel, Motion and Time Study Principles and 
Practices, New York, Prentice-Hall, Ine., l » Oe By 


(from manuscript). 


2. “How They Do It", Modern Management, Vol. 6, No. 4, 
May 1 s pp . L719. 


5. M. E. Mundell, yp Thoma e Motion and Time Study, Now 
York, Prentice-Hall, Ine., 1947, p. ° 
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worker satisfying all the above conditions and working sat 
such a standard pace. The mechanics of comparing ea given 
worker's pace to standard pace is called rating. It is here 
that the rater's judgment must play an important part as no 
precise, mathematical way of comparison has yet been suc- 
cessfully devised.*» © nis phase of time study, rating, 
should be the strongest link in the chain because its ve- 
racity and fairness must be apparent if the standard time 

eo produced is to be accepted with trust and confidence by 
all concerned. 

There are many systems of rating, nearly all relying 
entirely on judgment. How well this judgment is used, what 
basis it rests upon, and the manner in which it takes into 
account the various factors of skill, aptitude, and degree 
of exertion are extremely important quantities in making a 
consistent, reliable and fair rating. Many systems heve 
evolved using an estimate of effort to which individual 
company fatigue, personal time, tool repair, etc. allow- 
ences are added. 

The actual item which must be judged is pace or rate 
of activity. The standard rate of activity on a particu- 
lar job is chosen as that pace desired to be maintained. 


There are many numerical designations of standard pace 





4. M. BE. Mundel, Op. Cit., p. 155. 


5. L. Pe Alford and John R. Bangs, Production Handbook, 
New York, The Ronald Press co., 1947, pp. 505-507. 
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such as 60, 78, 80, 100, 120, ete. Fach has its own peculiar 
means of manipulation to assist in arriving st a standard 
time. 

Just how well industry is accomplishing the task of 
rating is a primary factor in the entire success, accepte 
ance, and use of time study. It is ea matter of importance 
to management and labor alike. As one prominent trade 
unionist states it; "Thus the solution to the basic problem 
of the validity of existing time study practice lies at the 
very heart of satisfactory industrial relations. "© Thus, in 
just one example of the uses of time study, management and 
labor bargaining collectively, such bargaining should not 
and must not rest on a chance basis of inaccurate and in- 
ept ratings and their attendant poorly allocated standard 
times. Similarly, the many other uses of time studies must 


not rest upon any insecure or wavering besis. 





Se William Gomberg, A Trade Union Analysis of Time Stud 
Chicago, Science Research Associa 6s, 1948, p. 15. : 
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OBJECTIVE AND PURPOSE 


The objective of this thesis is to gather and array 
data on current practice time-study ratings in such a man- 
ner that will permit of analysis as to consistency, accuracy, 
and amount of variation. These three factors in the rating 
of job pace or rate of activity are extremely important 
necessities for the widespread acceptance of and confidence 
in time study work. Just how well does present practice of 
time study engineers deserve such confidence and acceptance? 
It is hoped that a partial, though limited, answer to this 
question may be found in this paper. 

Many experienced time study engineere have stated that 
estimates of pace rarely vary more than five percent.’ How 
reliable is this estimate of raters’ abilities? Are manege- 
ment and labor justified in taking action based on such an 
estimate of consistency? 

it was further desired to ascertain what outstanding 
differences, if any, might be found with regard to the fol- 
lowing perameters: (1) years of experience of the rater; 
(2) concept of standard individually used; (3) type of 
training initially received; (4) size of rater's own plant; 
(6) geographic area in which the rater's plant is located; 
and (6) the size of town or city in which the individual 


rater's plant is located. Taking into account the region 





7. Dale Yoder, Personnel Management and Industrial Rela- 
tions, New York, Prentice-Hall, Inc., 1948, p. 143. 
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drawn upon, and in the hope of most logically uncovering the 


existence of or lack of any differences, the above parameters 


were subdivided as follows; 


(1) 


(2) 


(3) 


(4) 


(5) 


Years Experience 

(A) less than six months 
(B) 6 months - 2 years 
(C) 2- 4 years 


(D) more than 4 years 


Concept of Standard Used at Plant 
(A) your own concept of standard performance 
(B) some film or other embodiment of standard 


performance, 


Initial Time Study Training Received 
(A) Company 
(B) College 


Size of Plant 

(A) O = 200 employees 

(B) 200 - 1,000 employees 
(C) over 1,000 employees 


Plisnt Geographic Location 

(A) Northern Midwest (less Michigan) 
(B) Central Midwest 

(C) Southern Midwest 

(D) Miehigan 
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(6) 


Size of City, Plant Located in 


(A) 
(B) 
(Cc) 
(D) 
(E) 
(F) 


O - §,000 

5 - 10 thousand 
10 = 25 thousand 
25 - 50 thousand 
50 - 100 thousand 
over 100,000. 





PROCEDURE A 
PREPARATION FOR AND RECEIPT OF RATINGS 


Six films were obteined from a large southern Indiana 
industrial concern. Each film consisted of several cycles 
at different paces of a particular, actual incustrial job. 
After a careful check for spurious motions, constancy of 
pace, and of cycle times, three film loops were cut from 
each film. These cighteen film loops, each of exactly one 
cycle, became the jobs which were to be rated. No felse 
experimental conditions are thus set up because previous 
research has shown that time atudy engineers can rate pace 
from films or by actually observing the worker with prace 
tically equal facility.® These jobs covered the range from 
simple finger, hand, and arm movements of light assembly 
and inspection to the gross body movements used in load- 
ing and operating a large forming press. 

The next necessary component, the raters themselves, 
were the participants at the Fifth Annual Motion and Time 
Study Work Session (1950 session) held at this university. 
These seventy-two raters were from throughout the Middle 
West and nearly all were actively engaged time study engine- 
ers. 

A short briefing of the assembled engineers was made 


prior to the showing of the films. It was emphasized that 
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the various jobs and paces of these jobs were to be rated 
exactly as if they were occurring and being rated in the 
rater's own plant. Ratings were assigned according to the 
rater's own company standard and company policies and were 
then converted to a common, comparable, numerical besis by 
each individual rater. The common basis selected was the 
100 standard-150 maximum non-hermful physical effort defi- 
nition. The raters were furnished a simple, arithmetic 
conversion table to convert their own rating scale, no mat- 
ter what standard, to this scale. 

The work session was cautioned not to rate the method, 
to assume the method used the best for the particular job 
in question. No method variations were given on any of the 
joba and the raters were told to assume that work of ac- 
ceptable quality accompanied each of the eighteen performe- 
ances. 

The assemblage was further cautioned that although 
there were three paces each of six different actual indus- 
trial jobs, there was no set ratio between paces on the 
same or different jobs; and that the three paces of each 
job, themselves, might be practically the same or quite 
different. 

Finally the films were shown, and in an entirely ran- 
dom order so that no set sequence of pace or job could 
possibly influence the raters. Although shown in random 
Order, the films are numbered consecutively from one to 


eighteen in this paper, each three numbers in order 
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referring to the three paces of a particular job. 

The information enabling the breakdown of the six 
major categories into their several subdivisions was ob- 
tained from personal data questionnaires filled out prior 
to the showing of the films.” 

At this point it is desired to state that the manner 
in which the ratings were recorded and tabulated has much 
to do with the size and scope of the analyses of this work. 
Each rater placed his converted ratings on an IBM marke 
sensing card which then later was punched and used to assist 
in tabulating the results. The facility with which the 
various ratings and paeremeters could be entered on these 
cards, the speed and accuracy of assembling all or any seg- 
ment of the data by any particular parameter or subdivision, 
and the complete tabular arrays, all produced by the vari- 
ous IBM equipments are an outstanding addition to this type 
of research. Long, tedious hours of data assemblage and 


arraying became short minutes to the electronic "mind". 





9. Bernard S. Borrus, The Present State of Time Study, 
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PRCCEDURE B 
ASSEMBLY AND COMPUTATION OF DATA 


A logical division of the procedure occurs at this 
point. The first phase included the preparation for and 
the receipt of the “current practice” ratings of pace by 
the group of time study engineers. The second phase con- 
sists of the proper assemblage, manipulation, and computa- 
tion of the data necessary to form the basis for any pos- 
sible results and conclusions. 

The individual ratings assigned to a particular film 
were next arrayed for the whole group (all 72 raters) and 
by the various subdivisions of the six parameters. For 
each array, the average or arithmetic mean was determined 
as the initial basis for comparison. Consistency, the 
analysis of which was one of the primary objectives, is 
the measure of how the individual ratings of a given array 
ere grouped about its arithmetic mean. 

However, to call this "average rating” the actual or 
true rating, even though it is the concensus of opinion of 
72-0aa professional time study men, would be an untenable 
assumption. Much previous information on the subject has 
determined a general tendency to rate the slower paces too 
high and the more rapid paces too low,2° Therefore, a 


corrected mean, or rather, a best available approximation 
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of the correct mean reting, was computed with the aid of a 
precise knowledge of the ratios between cycle times for 
each of the three paces on a given job. A consistent series 
based on the actual cycle times for each pace was obtained 
for each of the six different jobs. The consistent series 
was adjusted by a saingle factor so that when this series 
end the averages of the individual ratings in eeach array 
on each job were then plotted, the sums of the squares of 
their deviations about Y=X were a minimum. A value, s0 
obtained, thus becomes the best approximation of the cor- 
rect mean rating, hereafter referred to as the corrected 
mean. Accuracy, whose analysis was another primary objec- 
tive, is the measure of how the individual retings of an 
array are grouped about its corrected mean. 

It 4s apparent that the term "array" has assumed an 
evereincreasing importance in this paper, reoccurring most 
frequently. To yield a clearer picture of what is refer- 
red to when thie word appears, the following is offered in 
explanation. The ratings for a particular pace Dy a par- 
ticular subdivision of a parameter, such as "2 - 4 years" 
of “Years Experience", are arranged from lowest to highest. 
This is an array. For each array the range, lowest sub- 
tracted from the highest rating; the arithmetic mean, the 
corrected mean, and various groupings about these means 
may be easily computed. The importance of arrays in this 
work may be further emphasized by the quick calculation 


thet, having eighteen paces and twenty-two subdivisions of 
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parameters, a resultant 396 arrays must necessarily be ar- 
ranged and manipulated. 

In order to carry out the other objectives, it be- 
comes necessary to find an additional besis for comparison 
between the various parameter subdivisions. This basis to 
be capable of consolidation of much bulk into a few statis- 
tics that will submit to further analysis. Thus, for each 
array, limits were set up at the 25%, £7244, %10%, and the 
520% levels about the actual means to indicate comperative 
consistency of rating, and similar limits were set about 
the corrected means for an indication of comparative ac- 
curacy. Percentages of each array lying within seach set of 
limits were then determined and are tabulated in the varie- 
ous Tables in the Appendix. The average percentage for 
each group alsaoO appears. 

Reference to any of the Tables mentioned above will 
show quite wide variations in percentages between the 
various subdivisions of a parameter. The next step was 
to determine if any statistically significant differences 
actually existed. A standard F test was made for signifi- 
cant differences at either the 5 or 1 percent confidence 
levels between means within a parameter. For those that 
indicated a difference in means greater than could be ex- 
peeted by chance alone, Fisher t tests between all com- 
binations of pairs within a parameter were made. Results 
of the t test at the 5% level indicated a reasonable doubt 


that the differences between two means could be due to 
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chance alone; while results at the 1% level indicated quite 
probably, i.e., 99 times out of 100, that the differences 
were due to some factor other than chance. The results of 


both the F and t tests appear in Tables 24 and 25 respec- 


tively. 
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RESULTS 


Probably the most important results appear in Tables 
1 and 2 of the Appendix. These portray the various find- 
ings for the work session taken as a whole. For consis- 
tency of rating, within 25% of the group mean we find an 
average of 38.2 percent, or nearly two out of five, which 
increases to 64.9 percent, nearly 2 out of 3, within 710%. 
The results for overall group accuracy are somewhat lower, 
ranging from 1 in 3 within the 45% limits to nearly 3 out 
of 5 within +10% of the corrected mean. 

fhe average range of the ratings of pace for the 
whole group, still based on 100 as standard, appears in 
Table 2 as 77 - 154, exactly a 2 to 1 spread. The average 
rating to which such an average range applies is 113 per- 
cent. 

Also found in Table 2 are the eighteen arithmetic and 
corrected meana and the variations between each, runging 
from O on film 15 to 12.9% on film 9, showing an average 
variation of 6.2%. 

Tables 3 through 23 show the percentages within the 
limits of their respective means for the various subdivi- 
sions of the six parameters. From these tables, with the 
aid of Tables 24 and 25, the following statistically sig- 
nificant differences between means are found; 

Initial Training. Between "Company" and "College" 


for both consistency and accuracy. 
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Area. Between Michigan and all other groups in con- 
sistency only. 

Size of City. Between over 100,000 and all other groups 
in consistency only. 

Between 5 - 10,000 and all groups but 
O - 5,000 in consistency. 

Between O - 6,000 and 25 - 650,000 in 
consistency. 

Because of the industry, attitude, and whole-hearted 
cooperation of the members of the 1960 work session, it 
is believed that the results and data compiled in this 
thesis are probably representative of time study practice 


in the Middle West area. 
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CONCLUSIONS 


1. The premise that tize study engineers’ ratings 
of pace rarely vary more than 5 percent is not borne out. 
Only 38.2 percent of the work session were within 154, and 
no subdivision had more than 56.7 percent within these 
limits of its mean. 

2. The average range of 77 - 154 on an average pace 
of 113 percent appears excessive in that all raters were 
viewing the same films at the same time. However, it 
should be pointed out that 89.7 percent of the whole group 
were within 220%, yielding a range of 90 - 135 (about 113) 
in which 9 out of 10 raters are found. 

S. The group as a whole exhibited a marked tendency 
to rate the faster paces more consistently than the slower. 
On the slowest pace shown (film 4 - 97 percent) only 17 
pereent of the work session participants were within 25% 
of the mean rating, while on a typical faster pace shown 
(film 12 - 134 percent) a total of 53 percent were within 
~5% of the mean. This effect continued with only 48 per- 
cent of the group within 510% of the mean rating on the 
slowest while 81 percent were within these limits of the 
mean rating on the faster pace. 

4. Throughout the analysis, aocuracy, or grouping 
about the best available estimate of the correct pace, was 
slightly inferior to consistency (the grouping with re- 


spect to the arithmetic mean pace), 
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5. Also apparent in the many errays is the previous- 
ly mentioned "regression" tendency to rate the lower paces 
too high and the higher peces too low. 

6. No significant differences at the 5% confidence 
level between the various subdivisions of the BXPERIENCE, 
CONCEPT OF STANDARD, and NUMBER OF EMPLOYEES categories is 
found. However, it is desired to point out that a compari- 
zon of Tables 7 and 8 will show that the raters having a 
“Film or other embodiment of standard" concept are both 
more consistent and more accurate within all rating error 
limit groups, having percentage pluralities ranging up to 
ll percent. 

7. Significant differences between means existed at 
the 5% confidence level (1 time) and at the 1% level (5 
timea) for consistency between "Company" and "College" 
under category INITIAL TRAINING (see Table 25). In each 
instance "College™ is lower than "Company" when comparing 
actual percentage means. It is not believed that the in- 
ference that receiving initial time study training in col- 
lege is inferior to company initial training can be sus- 
tained from this one category breakdown analysis, but 
rather that the possible interaction of other parameters 
such as years of experience, size of group, etc. have 
played an important part in yielding theece results. 

6. Under the AREA parameter, significant differences 
at the 1% confidence level were found between "Michigan" 


and all other subdivisions for all groupings. Reference 


wm 
~suatverg #0) 02 efor qmem woe ot tuteeqe ooin 
cet tod wens det edt Bre detd ood 
aoneblineo RS add de esoneisi2lb daveilingle om 6G 
| BOMAIRESAR ont AY anon tulbdue swwltey eid amewiet fave 
ef esiqgupeven BRAT2IT WO MAU Ore 2A ATR WH EESNOD 
-isegeoo 2 tact dup datog o¢ hoxrkeeb ef dt ,cevewell ofc? 
A metres avezet ode Sarit wore Lh iw 6 ine V sedset Qe ame 
sded sag Jqmedco "btebaate tu trwatbodae wate 15 mf PT” 
wovw guided ite adele atesuene oie Bos tuntetenco ewe 
~/ eee = oe OW Pee _—— wo teaerwed def 
da Deseizne sne0m weeied seonri@ltih tase igi OT 
 B) dovel Mi edt ap baw omtd L) Invel somsbitaoo Be ats 
“9B OJ Len" pae "EaegmGd" meewsed Pametetedoo wt (seat? 
toes of -( 88 pide e008) OATHLOAT aaIQidT yunpetso tenew 
axisameeo cade "yaeqoo” ages tench et Soysl£ol" seumdent 
-at ea? fect bevalied goa ef 4 .Ramem pantnantag Lawton 
-Len a4 gaiatat yudle omit imivint patvisows suds eomerel 
“em @¢ ceo yetutedd Isivini qmaquon ot iotstetet of ages 
fed ,2tetiana apobise vd Grepedae ewo eldt ant? beniad 
ezeiemeteg weido Fo Wottomiedal sidQeee) ace ims wedded 
even .oc@ .quetye le evie ,eome las te Staaf aa come 
efimget sends goiblety 22 samy dewtieast we Soyet7 
eeore7stiih gqaaotdilngis ,1etamere¢ SHA off reba! 48 
“uegiie ti” ceewted Uno t o18~ invel eombivaos OL ety ta 
aace ow = oagriquom Lhe wo? sacks teglvwwe  wedtfo {le bre 











18. 


to the tables of sctual percentage moans shows "Michigan" 
possessing markedly lower averages throughout. However, 
the results here are not an indictment of the whole 
"Miechigan” area as the sample size of only nine men is very 
small and two of this subgroup assigned extremely high 
ratings throughout the entire eighteen films, 

9. Under SIZE OF CITY the raters from "Over 100,000" 
are significantly and consistently below all other groups. 
Likewise are the "5 - 10,000" raters below nearly all 
others. Again, it is believed thet probably the extremely 
smell sample sizes of these two groups and the interaction 
of other factors preclude the definite conclusion that in- 


ferior ratings correlate with such sizes of cities. 


AEB OLe” acto wedem SYRd Teey [eaten To andes eid ot 
gg PORUTER Th etOUNT RegutevA sewol yibeaiam yrlesee co 
«foie ert To tmebstiot ae tee ere peed ec igen: ode 
‘crsv) el Gee onl pie to erin siysme seid ee aterm "mgt rio ie” 
ye Cameases hemyeee eeurgawe efit Le awe bre [imme 
he tate SETAE AeNiREO TS Betton 
"oet,2Of reve" wort eveten odd YOED W) €5Te weet 62 
evo wacto Lfe woled Eidrwés taneo hee (LomvetPieyte ecm 
«the Eltwem woled exever: "G0 Ql + Bf ett e498 08 test 
Phedeides eit Eldadene dave bevelled ef 21 ~nlegh .atedio 
selégevetat etd baw oqu0t, od awema tc fostc olqume [ines 
“Al Saas noleulovon etinMes efd eGuloea) e10d0a% tetito Yo 
gy iete re seats fees dziW Sta feerso egutder 1olttt 


= a a . = gy a — ~ ies oe -* 
-_— ‘ ammmel ro me gen —7~ ee. * ee — + om! + 

— « —_— sds = a 7 —— oe 
- © &e« -— <™ i ma o- ——— ~ Soe «ee eee 


ak . _ : 0 > dene — 


19. 


INFERENCES 


1. It is felt that this analysis of “current prac- 
tice" time-study rating shows that the present state of 
time-study rating is not up to the level required for the 
widespread confidence and acceptance that is so necessary 
for its use. The need for time studies and their proven 
aid in solving many complex industrial problems have been 
demonstrated for many years. The need now is for more con- 
sistent, accurate, and reliable ratings of pace and their 
end product, more absolute, trustworthy, and authoritative 
standard times. 

2. Perhaps the single most important inference that 
can be gained from this analysis is that there exists an 
urgent need for removing or minimizing the personal “judg- 
ment" factor from the individual rating. The wide ranges 
of ratings when viewing the same rates of activity must be 
lessened if we are to achieve more consistent ratings. 

The use of different concepts of atandard pace and 
the reliance on personal judgment must be minimized. [In 
other words, a more concrete basis must be found for the 
rating procedure. It is believed that the exception noted 
under Specific Conclusion 6, while not conclusive in it- 
self, points the way to possible ultimate reduction to 
insignificance of the judgment factor. It is suggested 
that the use of a film showing a standerd rate of activity 


or various percentages thereof, shown adjacent to or 
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superimposed upon a film of the pace desired to be rated 
can result in much more consistent, eccurate, and reliable 
ratings of pace by giving the rater a visible, concrete 
basis on which to exercise the then modicum of personal 
judgment required. While this procedure may not be ec- 
Onomically feasible for the making of all job pace ratings, 
a judicious use of such a procedure should, in all prob- 
abilities, result in a definite minimizing of the judgment 


factor. 
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Table l 
ENTIRE GROUP - 72 RATERS 








Consistency 
Film Arithmetic Percent of Group within 
No. Meen Rating 75% Must 4 710% +20¢ 
1 111 ey 51 58 87 
2 112 41 57 71 88 
5 110 38 §1 60 88 
< 97 18 35 48 84 
5 113 32 58 54 88 
6 136 §1 62 76 93 
7 101 $1 43 59 90 
8 107 41 50 66 88 
9 101 25 38 47 78 
10 110 40 51 59 91 
li 118 bot o) 56 65 be 
12 134 535 65 81 93 
13 116 40 60 68 93 
14 113 +4, §3 68 90 
15 114 43 61 68 04 
16 112 35 64 74 90 
17 117 40 61 69 93 
18 117 $1 72 78 9S 
Average 58.2 63.2 64.9 89.7 
Accuracy 
Film Corrected Percent of Group Within 
No. Mean Rating £52 £72% 2104 +20% 
1 104 29 43 53 82 
- 105 37 50 57 85 
3 122 19 34 47 76 
4 8&7 26 o2 44 72 
5 109 35 4’? 57 90 
6 144 59 51 69 87 
7 90 Le 32 44 63 
8 101 25 40 54 85 
9 116 12 24 36 64 
10 104 29 “9 §7 88 
11 112 55 54 67 89 
12 143 ed 40 65 94 
13 108 28 39 §7 9% 
14 120 44 50 60 88 
15 114 43 51 68 94 
16 107 47 58 68 87 
17 115 44 56 68 93 
18 115 53 68 82 94 


Average 33.1 45.4 58.5 84.6 
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Average 


COMPARISON OF ENTIRE GROUP MEANS AND RANGE 


Range 
65 - 156 
70 = 149 
65 - 161 
62 - 143 
75 - 162 
85 - 169 
50 = 143 
65 - 149 
75 - 1435 
78 - 181 
87 «= 156 

102 = 178 
81 - 161 
78 = 162 
66 - 161 
76 = 149 
87 - 156 
92 = 156 
77 - 154 


Table 2 


Arithmetic 


Mean Rating 


lll 
112 
110 

97 
115 
156 
101 
107 
101 
110 
118 
134 
115 
1135 
114 
112 


Corrected 


Mean Rating 


104 
105 
102 

87 


Percent 


Variation 
6.7 


6.7 
9.8 
11.5 
5.7 
5.5 
12.2 
5.9 
12.9 
5.8 
5.4 
6.5 
7.4 
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Film 
No. 


Film 
No. 


arithmetic 
Mean Rating 

107 
114 
107 
100 
115 
128 

97 
106 

92 
113 
109 
131 
110 
108 
116 
1138 
114 
114 
Average 


Corrected 
Mean Rating 
102 
104 
120 
67 
108 
142 
85 
95 
110 
101 
109 
140 
106 
117 
lll 
106 
114 
114 


Average 


Table 3 
EXPERIENCE 
LESS THAN 6 MONTHS - 6 RATERS 
Consistency 
Percent of Group Within 
25% = 78% 210% 
33 67 67 
33 67 67 
Me Pe) 50 50 
0 @) e) 
17 17 50 
17 50 67 
53 oO 50 
ro Fe) 50 50 
50 50 67 
17 17 17 
50 83 100 
17 33 67 
30 60 67 
SO 67 83 
50 50 67 
33 67 67 
67 67 67 
50 67 83 
D5ed 49.2 60.3 
Accuracy 
Percent of Group Within 
£54 ~ 73% ~10% 
50 67 67 
17 ood ro) 
17 by Po) 50 
17 17 30 
33 33 DO 
50 50 67 
0 0 @) 
17 17 17 
17 17 335 
17 oho) Po) 
50 83 100 
38 33 50 
OO 67 83 
50 67 67 
33 50 50 
33 33 33 
67 67 67 
50 87 83 
3204 42.6 50.0 
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Table 4 





EXPERIENCE 
6 MONTHS - 2 YEARS - 16 RATERS 
Consistency 
Film Arithmetic Percent of Group Within 
No. Wean Rating 254 = 78h 210% =20% 
1 112 44 75 75 04 
2 114 4% 56 81 94 
3 111 37 56 63 94 
4 98 19 31 50 94 
'e) 118 25 31 69 Y= 
6 138 50 69 é9 8& 
7 104 26 63 69 88 
8 109 44 56 635 94 
€ 101 25 38 50 88 
10 112 25 ol 50 68 
11 117 38 69 69 Ga 
12 137 4 56 81 94 
138 116 25 63 63 04 
14 115 31 “4 63 88 
15 115 56 56 81 88 
16 113 S1 a4 63 100 
17 117 25 50 50 g4 
18 116 50 63 WES) 94 
Average 55.4 52.8 65.8 92.7 
average 
Film Corrected Percent of Group Within 
1 105 50 63 63 34 
2 107 37 50 56 94 
3 123 19 37 50 81 
4 89 31 50 56 81 
5 112 50 56 63 88 
6 148 44 56 63 88 
7 91 25 58 44 69 
8 102 13 31 66 88 
9 117 6 25 ol 65 
10 105 25 50 56 88 
11 113 44 56 75 24 
12 145 19 44 63 100 
13 109 13 31 50 83 
14 121 38 56 56 83 
1& 115 56 56 81 100 
16 107 ol 63 69 83 
17 115 31 38 50 100 
18 115 56 63 WA?) 100 


Average 32.7 47.9 58.7 88.1 
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Table & 
EXPERIENCE 
2 YEARS - 4 YEARS - 19 RATERS 
Conai stency 
Arithmetic Pereent of Group Within 
Mean Rating =5% 72% ~1L0% 220% 
111 42 47 47 65 
113 50 50 88 34 
109 ae 69 69 100 
96 19 19 25 75 
lil 25 50 56 100 
156 68 68 84 100 
100 19 25 56 88 
104 25 56 81 88 
106 32 57 58 90 
108 50 56 81 100 
120 47 47 65 100 
134 74 79 100 100 
116 55 68 84 100 
112 50 75 81 100 
109 o8 a4 50 100 
110 74 84 90 95 
118 47 74 95 100 
118 53 79 84 95 
average 45.0 §7.1 71.8 93.7 
accuracy 
Corrected Percent of Group Within 
Mean Rating 25% = 78% 210% +20 
104 42 47 47 68 
106 37 50 635 88 
122 19 44 63 81 
87 6 6 19 56 
109 58 44 56 100 
143 42 47 84 100 
90 25 26 ol 50 
101 25 38 63 94 
116 26 26 §3 79 
104 44 63 75 100 
112 26 42 §3 84 
Lae 21 42 90 100 
107 38 42 58 90 
118 50 69 75 100 
112 31 ge 56 100 
107 63 74 90 95 
116 58 68 84 100 
116 42 84 90 96 
average 56.2 47.5 635.9 87.8 
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Table 6 
EXPERIENCE 
MORE THAN 4 YEARS - 29 RATERS 
Consistency 
Film Arithmetic Percent of Group Within 
No. Mean Rating 25¢ = TRS 210% ~ 20% 
1 109 31 41 62 90 
2 109 54 61 82 96 
3 106 39 50 57 93 
4 93 43 54 61 93 
5 108 52 46 61 89 
6 133 45 65 76 ° Bs 
7 100 32 54 57 93 
8 106 359 50 68 93 
9 98 24 31 55 83 
10 107 36 57 71 93 
Ld 115 35 55 79 90 
12 132 62 69 72 gO 
13 115 52 55 69 97 
14 111 39 46 68 89 
16 114 43 54 71 96 
16 110 45 62 69 93 
17 116 48 69 72 86 
18 116 59 72 8S 95 
Average 42.1 54.5 68.0 91.7 
Accuracy 
Film Corrected Percent of Group Within 
No. Mean Rating 25% t78¢ ~10% *20¢ 
1 101 “4 34 45 72 
2 103 46 61 64 89 
3 119 21 43 46 79 
4 84 29 39 54 79 
§ 106 32 50 64 89 
6 139 45 62 72 93 
7 88 21 56 46 68 
8 99 32 36 54 86 
9 114 7 21 Mo} o) 59 
10 102 25 59 50 93 
ll 110 52 59 66 86 
12 140 55 52 66 90 
13 107 31 45 §9 79 
14 119 46 57 64 82 
15 113 52 46 64 96 
16 106 45 52 55 86 
17 114 358 62 72 90 
18 114 62 76 76 97 


Average 34.6 48.3 58.4 84.0 
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Film 
No. 


Film 


NO. 


~ 


Arithmetic 


Mean Rating 
110 
115 
109 
96 
112 
1&4 
102 
107 
101 
110 
117 
134 
115 
112 
114 
lll 
117 
117 
Average 


Corrected 


Mean Rating 
103 
106 
122 

86 
108 
143 

89 
100 
115 
103 
Al2 
1435 
108 
119 
113 
107 
115 
118 


Average 


Table 7 


CONCEPT OF STANDARD 
OWN CONCEPT - 54 RATERS 


Consistency 
Pereent of Group Within 
272% 210% 


25% 320% 
50 56 61 89 
46 50 84 90 
36 54 EO 92 
24 350 42 76 
42 44 58 RE 
44 66 70 93 
32 56 60 90 
40 46 68 90 
30 41 52 85 
38 50 58 90 
52 61 V2 e3 
46 63 82 G4 
41 61 72 96 
40 54 66 90 
38 46 60 34 
57 74 78 93 
359 63 72 91 
54 a) 80 96 
40.5 54.4 66.4 90.8 
Accuracy 
Percent of Group Within 
26 44 48 87 
40 64 §8 86 
a4 30 48 82 
20 32 44 6& 
32 48 64 92 
41 46 72 91 
16 28 46 66 
28 36 56 86 
13 22 39 70 
28 48 58 8& 
45 52 67 91 
26 41 65 96 
32 39 56 83 
46 60 68 8&8 
30 44 54 94 
46 59 74 87 
a4 56 70 93 
50 KE 78 96 
32.6 44.9 59.2 85.8 


— —_, —_ —_ ——— 





FY sew amar mestadssee 





Eel waner eo tINeeser'd 








Table 8 


CONCEPT OF STANDARD 
FILM OR OTHER EMBODIMENT - 15 RATERS 


Consistency 
Film Arithmetic Percent of Group Within 
No. Mean Rating ~5% 274% 210% = 20% 
1 112 7 47 63 100 
2 107 353 53 73 100 
3 109 47 §3 73 100 
4 96 27 §3 67 100 
5 111 20 47 60 100 
6 1358 60 80 100 100 
7 101 7 40 60 100 
8 107 53 60 67 100 
9 100 27 80 47 93 
10 109 53 60 73 93 
11 115 ror] 53 73 100 
12 134 67 80 95 100 
13 114 60 67 50 100 
14 113 53 80 100 100 
15 112 60 95 93 100 
16 112 27 60 80 100 
17 115 53 67 73 100 
18 114 7 80 100 100 
Average 42.2 61.8 75.8 99.2 
Accuracy 
Film Corrected Percent of Group Within 
Wo. Mean Rating 25% t78% 210% 220 
1 102 63 53 60 67 
z 104 33 47 67 100 
Ps) 120 13 53 60 93 
4 87 40 40 47 80 
§ 110 47 47 60 100 
6 144 §3 73 93 100 
7 89 33 47 47 53 
e 100 20 53 67 93 
9) 118 20 27 47 67 
10 103 47 55 60 87 
11 111 27 53 60 87 
12 142 53 53 80 100 
13 107 20 40 55 93 
14 118 60 60 73 100 
15 112 60 93 93 100 
16 106 53 67 67 93 
17 114 Ss 67 73 100 
18 114 73 80 100 100 


Average 42.1 55.9 67.1 89.6 





Teble 9 


INITIAL TRAINING 
BY COMPANY - 46 RATERS 


Consistency 
Film Arithmetic Percent of Group Within 
No, Mean Rating 25% = 73% -10% =20% 
1 112 29 51 58 89 
2 114 51 76 85 90 
3 110 ad 61 73 95 
4 97 22 37 a4 88 
§ 113 37 46 63 93 
6 134 58 73 89 96 
7 103 27 4) 56 90 
8 109 56 68 78 90 
) 103 20 40 4, 89 
10 112 44 63 63 93 
li 118 58 §8 71 v8 
12 135 53 67 84 96 
13 109 47 71 87 98 
14 121 54 61 78 93 
15 115 46 56 78 95 
16 112 44 78 84 93 
17 118 47 73 87 98 
18 118 58 82 84 95 
Average 43.0 61.6 72.8 93.1 
ccurec 
Film Corrected Percent of Group Within 
No, Mean Rating 25% 73% 210% ~20% 
1 105 45 47 60 82 
£ 107 41 54 66 90 
Fo] 127 20 32 $9 90 
€ 87 22 29 39 68 
& 109 57 51 63 93 
6 1438 44 51 76 96 
7 91 27 49 56 90 
8 102 32 68 78 90 
9 118 16 40 49 89 
10 105 32 51 66 90 
Li 113 42 49 69 93 
12 144 20 38 67 98 
13 118 38 68 69 93 
14 114 354 54 63 98 
1& 115 46 56 68 95 
16 108 56 64 78 93 
17 116 60 71 80 OS 
18 116 56 82 89 93 


Average 43.0 52.4 66.3 90.7 
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Table 10 


INITIAL TRAINING 
IN COLLEGE - 22 RATERS 


Consistency 
Film Arithmetic Percent of Group Within 
No. Mean Rating 25% +7a% =10% 20% 
1 lil 45 45 "68° 91 
2 111 32 45 41 82 
3 111 41 41 45 Be 
4 98 18 32 50 62 
5 116 9 32 45 82 
6 142 45 64 75 82 
7 101 56 50 59 91 
8 106 27 41 55 66 
a 99 32 32 64 86 
10 110 32 41 50 91 
11 118 32 55 55 86 
12 136 45 68 77 ol 
13 115 Lo S2 50 86 
14 114 §5 59 68 91 
15 115 41 45 64 91 
16 113 Mey 7 56 65 &6 
17 117 32 45 68 91 
18 116 45 59 64 95 
Average 5425 45.6 68.3 67.1 
Accuracy 
Pilm Corrected Percent of Group Within 
No. Ween Rating 25 72% ~10% = 20% 
1 104 56 56 50 82 
2 106 36 4§ 45 73 
3 122 18 41 50 73 
4 89 23 06 45 77 
5 113 27 41 45 82 
6 149 41 655 55 82 
7 89 14 oe 45 64 
8 106 27 41 55 86 
3 114 18 18 56 64 
10 104 32 56 50 91 
ll 112 23 50 59 82 
12 144 27 50 64 BT 
13 109 18 36 50 86 
a 120 55 64 73 77 
15 114 44 4§ 68 Gl 
16 107 2 45 55 77 
17 115 27 41 50 Ol 
18 115 45 55 73 95 


Average 0.0 39.8 63.7 81.2 
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Film 
Noe 


Arithmetic 


Mean Rating 
102 
104 
103 

83 
102 
126 

92 
101 

91 

97 
108 
128 
110 
108 
107 
107 
109 
113 


Average 


Corrected 


Mean Rating 


96 
98 
113 
79 
99 
130 
82 
92 
LO6 
96 
104 
135 
103 
114 
108 
102 
110 
110 


Average 


Table 11 


NUMBER OF EMPLOYEES 
O = 200 - 10 RATERS 





Consistency 
Percent of Group Within 
t5¢ 27a% 110% 220% 
80 90 90 100 
10 Ko) @) 30 50 
10 50 60 90 
40 70 90 90 
40 60 €0 100 
40 50 70 90 
20 40 40 80 
10 30 60 80 
40 50 50 90 
50 70 80 100 
Py @) 50 90 100 
60 80 80 100 
40 70 90 100 
30 70 80 100 
40 40 60 100 
50 70 80 90 
20 40 60 100 
40 50 90 
56.7 66.1 71.1 93.9 
Accuracy 
Percent of Group Within 
25% 278% 2102 220% 
20 <0 80 169 
£0 30 30 &0 
50 50 60 90 
60 80 90 SO 
20 40 EO 100 
£0 50 70 90 
10 30 50 50 
30 rol @) 3O 80 
20 20 40 70 
50 70 80 90 
30 70 70 100 
60 80 90 100 
50 60 60 100 
40 50 60 100 
40 50 60 100 
50 60 70 100 
20 40 70 100 
50 60 70 100 
56.1 50.90 64.4 90.6 


SEj-wescer end t¥eesere 





fYpraeseer se dlHss3se2 








isSeegeiz at [psbaiosntiei| 


Meese sssgsesssy sey 


me 





SSRSSSESERSE a95308 





SSbBSsEE Sie 





Table 12 


NUMBER OF EMPLOYEES 
200 = 1,000 - 37 RATERS 


Consi atency 
Film Arithmetic Percent of Group Within 
_No, Mean Rating 55% 273% 210% 220% 
1 111 41 52 57 84 
2 114 43 635 77 89 
x) 111 28 4.6 65 86 
4 98 26 54 46 83 
5 114 37 46 54 86 
6 1355 57 65 76 93 
7 103 25 1 §7 89 
8 108 43 51 71 [2 
9 101 24 35 44 79 
10 112 43 54 63 92 
11 118 53 54 71 95 
12 135 44 54 79 90 
135 116 52 71 74 95 
14 113 37 60 66 86 
15 115 43 49 71 66 
16 112 35 54 65 93 
17 118 41 68 B34 "1 @) 
18 117 52 76 82 95 
Average 39.0 5446 66.7 87.8 
Accuracy 
Film Corrected Percent of Group Within 
No. Mean Rating 26% t7h4% 1024 1208 
1 104 13 35 49 84 
2 106 40 57 60 86 
3 123 28 40 43 83 
4 87 17 26 57 71 
6 110 ol 40 46 86 
6 144 41 57 71 87 
7 90 17 ron | 43 54 
8 101 26 40 54 86 
9 116 9 22 33 68 
10 105 28 43 60 Oe 
11 113 94 63 61 90 
12 144 24 41 46 90 
13 109 50 52 65 90 
14 120 46 51 63 89 
15 114 435 49 69 era 
16 108 35 49 §7 90 
17 116 49 68 74 90 
18 116 57 76 84 90 


Average S2.1 46.7 57.0 84.4 





Table 15 


NUMBER OF EMPLOYEES 
MORE THAN 1,000 - 21 RATERS 


Consistency 
Film arithmetic Percent of Group Within . 
Noe Mean Rating 25% =74% 210% =20% 
1 lll 19 48 62 100 
2 110 53 58 68 100 
5 106 37 68 74 95 
4 96 S2 58 68 100 
5 11il 37 63 65 100 
6 137 67 71 91 100 
7 v9 58 63 79 100 
8 106 47 58 68 100 
S] 103 58 4s 48 96 
10 110 58 74 74 956 
11 117 43 62 67 95 
12 132 81 86 95 100 
13 114 boo) 38 57 100 
14 111 47 79 79 95 
15 112 58 74 74 90 
16 111 48 62 81 95 
17 116 52 67 71 100 
18 115 87 76 86 95 
Average 48.1 63.8 72.5 97.5 
Accuracy 
Film Corrected Percent of Group Within 
No, Mean Rating 25% 274% 210% = 20% 
1 102 43 43 48 71 
a 104 42 53 68 100 
3 120 21 32 47 84 
4 87 47 47 53 84 
§ 109 63 63 79 100 
6 143 48 §7 86 100 
7 89 21 37 58 79 
8 100 37 63 68 95 
9 115 14 oP) 57 71 
10 103 37 47 63 84 
11 111 24 52 67 90 
12 142 43 43 81 100 
13 107 ag 38 48 81 
14 118 47 53 74 90 
15 112 58 74 74 90 
16 106 62 67 76 90 
17 114 48 §7 67 100 
18 114 _57 _76 _86 95 


Average 40.3 61.9 66.7 89.2 
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Table 14 
AREA 


NORTHERN MIDWEST (Less MICHIGAN) - 20 RATERS 


Arithmetic 


Mean Rating 
108 
110 
108 

91 
110 
132 

94 
105 
103 
106 
118 
134 
116 
117 
117 
113 
118 
117 


Average 


Corrected 


Mean Rating 


101 
103 
119 

54 
106 
139 

68 

99 
113 
103 
dil 
142 
lil 
123 
116 
108 
116 
116 


Average 


Consistency 
Percent of Group Within 


25% 278% =10% 220% 


35 60 80 8&5 
46 55 65 85 
45 60 70 90 
25 40 50 85 
4S 55 66 95 
05 55 70 85 
3§ 40 50 75 
40 50 55 75 
20 55 65 85 
25 35 70 90 
55 65 75 95 
45 60 70 90 
55 75 80 96 
45 55 65 95 
45 60 70 100 
40 50 76 95 
50 75 80 100 
65 15 80 100 
40.6 56.7 68.6 90.0 
Accuracy 
Percent of Group Within 
25% 273% 210% ~ 20% 
30 o§ 45 96 
50 40 50 80 
25 45 50 85 
15 35 50 76 
25 40 75 95 
45 55 70 85 
15 15 40 65 
15 20 28 70 
20 25 40 80 
15 45 50 90 
30 45 65 90 
35 45 60 90 
355 40 65 95 
40 75 80 95 
40 60 70 100 
45 60 70 90 
50 70 85 100 
60 75 80 100 
51.7 45.8 59.4 87.7 
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Table 15 
AREA 


CENTRAL MIDWEST - 31 RATERS 


Arithmetic 


fon 

109 
110 
105 

96 
108 
132 
104 
107 
101 
108 
115 
151 
113 
108 
111 
108 
113 
113 


Average 


Corrected 


Mean Rating 
101 
102 


Average 


sister 


Percent of Group Within 
5% 276% =10% 


t¢ 


4 52 55 94 
59 59 63 96 
o4 44 52 96 
00 he) @ As 89 
37 48 59 81 
55 68 74 100 
26 59 "63 100 
53 45 70 100 
‘ee 26 42 77 
26 41 63 96 
42 52 71 97 
65 78 84 97 
te 52 61 97 
35 41 67 93 
48 63 65 100 
o5 o9 45 55 
o& Moh”) &2 97 
42 58 90 100 
539.5 49.7 62.1 92.5 
Acgouracy 


Percent of Group Within 


05 45 58 71 
o7 62 59 96 
o7 48 52 85 
19 26 41 67 
41 48 63 65 
48 74 61 97 
22 26 Oo 55 
19 37 52 96 
ig 29 45 58 
4) 48 56 82 
og 52 71 87 
o§ 65 78 97 
2 de 42 61 
41 56 70 85 
41 63 70 100 
25 29 o5 52 
25 o§ 48 97 
48 58 68 100 
54.0 46.3 56.2 82.8 





fable 16 


AREA 
SOUTHERN MIDWEST - 12 RATERS 
Consi stency 
Film Arithmetic Percent of Group Within 

No. Mean Rating 25% 78% 210¢ z 

‘k 115 25 25 58 

2 113 50 50 67 

be) 115 50 58 58 

4 101 55 42 42 

§ 116 8 25 ae 

6 145 42 75 76 

7 102 8 25 Koka) 

8 108 50 58 75 

+) 99 17 17 33 
10 113 25 25 75 

ll 119 VO 33 42 
12 156 58 83 83 100 
13 118 ree) 53 75 
14 116 67 835 83 
15 115 42 L oa 67 
16 115 42 50 83 
17 121 50 83 83 100 
18 121 25 67 7 100 
Average 56.6 50.0 635.8 
Accuracy 
Film Corrected Percent of Group Within 

No. Mean Reting o5% = 74% 210% 2 

1 107 25 50 50 

2 109 50 58 67 

Fos 126 17 17 4a. 

4 92 33 42 42 

5 115 17 17 4- 

6 152 42 42 58 100 
7 90 42 42 58 

8 100 42 67 75 

G 115 C 17 33 

10 106 67 67 786 
1l 114 53 33 75 

12 146 17 42 42 100 
13 110 26 OO 50 
14 122 25 58 75 
156 116 35 50 67 
16 lll 42 67 75 

17 119 67 67 92 100 
18 119 58 87 15 100 


Average 36.3 46.4 60.7 








fable 17 


AREA 
MICHIGAN - 9 RATERS 








Film Arithmetic Percent of Group Within 
No. Mean Rating £5¢ = 74% 210% 20% 
1 120 22 22 44 89 
t 127 22 4% 56 100 
3 119 11 22 44 78 
4 108 0 11 44 78 
5 129 ll Ko Re) ae 100 
6 141 22 44 56 89 
7 108 22 ge 44 78 
8 113 33 44 56 67 
) 101 22 56 56 78 
10 121 33 56 56 67 
ll 121 44 44 44 89 
12 145 22 ee 44 100 
13 120 22 o3 56 100 
14 115 33 56 56 67 
1&5 117 Gs 56 56 67 
16 119 4 é, 44 56 89 
17 124 22 56 56 89 
18 126 £2 22 ss 78 
Average 25.1 59.4 50.0 83.5 
Accuracy 
Film Corrected Percent of Group Within 
No. Mean Rating 25% = 74% =10% t20% 
i 114 53 oh 44 78 
2 116 22 67 67 78 
3s 1354 ll 44, 44 67 
4 95 22 33 53 56 
6 119 11 11 56 78 
6 157 22 22 3S 78 
7 93 56 56 56 78 
8 106 ae 67 67 78 
© 120 11 11 ll 44 
10 111 35 33 56 78 
11 120 44 44 ad 89 
l2 163 £2 22 33 100 
13 111 44 44 a4 78 
14 123 22 33 53 89 
15 117 44 56 56 67 
16 113 22 aé 56 89 
17 121 33 56 56 89 
18 121 35 44 44 67 


Average 20.4 40.6 46.3 76.7 
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Table 18 


SIZE OF CITY 
OQ + 5,000 - 17 RATERS 





sonsistenc) 
Film Arithmetic Percent of Group Within 
No. Mean Rating r5% 572% 410% 220% 
L 112 35 59 59 94 
Z 100 65 71 77 8& 
3 109 53 65 62 94 
4. 4 41 41 47 94 
5 106 24 4) 5S 4 
6 1356 63 77 88 94 
7 100 24 2 65 94 
8 105 47 §9 65 68 
g 97 29 5S 65 82 
10 108 29 47 59 100 
1l 116 29 §3 65 100 
12 135 41 53 77 100 
13 115 71 71 88 100 
14 112 47 71 71 100 
15 112 29 69 65 100 
16 111 59 71 82 100 
17 116 47 65 77 100 
18 115 72 7 82 100 
Average 44.1 58.7 70.4 95.7 
Accuracy 
Film Corrected Percent of Group Within 
No. Mean Rating £54 74% +10% 120 
1 103 29 41 47 77 
. 105 55 47 65 82 
re) 121 29 47 71 8& 
4 85 18 29 41 77 
§ 107 o4 47 53 94 
6 141 71 82 82 [4 
7 88 18 18 35 53 
8 98 Ap 12 18 88 
i) 113 6 24 25 71 
10 103 6 41 47 88 
il 111 41 47 71 82 
de 142 41 47 71 94 
13 107 29 47 65 04 
14 119 47 71 77 94 
16 113 29 59 65 100 
16 106 53 65 71 Q4 
17 114 47 65 77 Ge 
18 114 71 71 77 100 


Average 33.7 47.8 §9.3 66.9 
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Table 19 


SIZE OF CITY 
5,000 = 10,000 + 7 RATERS 


Consistency 
Film Arithmetic Percent of Groupt Within 
No. Mean Rating 25% 272% 210% 220% 
1 109 43 57 71 71 
te 115 0 57 57 &6 
Mo) 113 29 43 57 86 
4 99 43 43 57 71 
5 122 29 43 43 66 
6 139 43 435 43 86 
7 98 29 he 29 57 
8 105 29 29 29 57 
9 109 71 71 71 100 
10 113 29 43 71 71 
11 124 29 29 57 86 
12 135 29 43 71 86 
13 lel 57 57 71 86 
14 122 29 71 71 86 
15 123 67 57 71 845 
16 115 29 57 71 86 
17 125 71 71 86 100 
18 122 1 86 8S 100 
Average 59.8 51.6 61.8 82.6 
Accuracy 
Film Corrected Percent of Group Within 
No. Mean Rating 26% = 78% =10% 220% 
1 106 43 67 67 71 
2 107 43 43 57 86 
3 123 14 29 29 71 
é 91 29 29 43 71 
§ 114 O 14 43 86 
6 150 43 43 43 71 
7 91 O 14 29 43 
8 102 14 Pag 29 57 
9 117 14 43 71 86 
10 107 29 43 57 71 
ll 115 43 §7 57 86 
12 147 29 43 57 86 
13 116 29 71 71 86 
14 128 29 29 29 86 
15 122 67 57 71 &6 
16 112 43 57 86 86 
17 121 57 86 66 100 
18 121 71 86 86 100 


Average 32.6 46.1 65.6 79.4 
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Table 20 


SIZE OF CITY 
10,000 = 25,000 - 19 RATERS 
Consistency 
Pilm Arithmetic Percent of Group Within 
No. Mean Rating 25% 272% ps ke) § s 
1 110 47 58 65 100 
2 109 60 60 67 100 
3 lll 47 67 73 100 
4 93 27 27 40 100 
5 107 47 60 80 80 
6 135 7+ 79 90 100 
7 103 27 53 60 100 
8 108 40 40 67 100 
9 103 16 26 26 100 
10 105 47 67 67 100 
11 114 42 5&8 90 100 
12 130 64 90 100 100 
13 116 47 58 74 100 
14 113 60 73 80 100 
16 112 47 60 67 90 
16 lil 63 68 74 100 
17 118 53 B4 95 100 
18 116 58 90 100 100 
Average 49.2 62.1 75.0 98.7 
Accuracy 
Film Corrected Percent of Group Within 
No. Mean Rating 25% = 74% ~10% =20% 
1 103 4c 47 53 79 
2 104 40 60 67 100 
3 121 27 53 67 87 
A 85 27 27 53 60 
5 107 47 60 80 93 
6 140 47 90 95 100 
7 91 20 40 47 67 
8 102 40 60 67 87 
3 117 16 26 38 58 
10 100 47 55 7S 93 
11 108 42 §3 87 95 
12 138 74 7% 90 100 
13 109 26 ay 63 106 
14 119 60 60 735 93 
15 113 53 67 87 100 
16 107 58 63 74 95 
17 115 a2 68 90 100 
18 115 58 90 100 100 


Average 42.6 68.0 71.4 89.5 





Film 
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Film 
No, 
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Arithmetic 


Mean Rating 
113 
114 
106 

95 
114 
136 

97 
111 

97 
1138 
118 
133 
116 
112 
113 
lll 
116 
113 

Average 


Correoted 


Mean Rating 


114 
104 
112 
144 
108 
119 
113 
106 
114 
114 


Table 21 
SIZE OF CITY 
25,000 ~ 0,000 - 10 RATERS 
Consistency 
Percent of Group Within 
£5 LTRS *10% 
60 60 70 
60 80 100 
70 80 80 
40 50 60 
70 70 70 
60 80 90 
70 80 90 
70 80 80 
60 70 80 
50 70 70 
70 80 90 
60 90 90 
20 70 70 
50 80 80 
40 50 50 
50 60 80 
70 90 90 
50 60 90 
56.7 72.7 79.4 
Acouracy 
Percent of Group Within 

25% = 72% 510% 

0 20 60 
40 50 60 
10 20 50 
30 350 40 
50 60 80 
50 60 80 
20 40 60 
40 40 40 
10 10 40 
40 60 80 
50 70 80 
00 50 70 
40 60 60 
50 50 80 
40 50 50 
50 70 80 
70 80 90 
60 80 80 
O7. 8 50 e 0 64 e 4 


Average 


100 


100 
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Table 22 


SIZE OF CITY 
50,000 = 100,000 - 7 RATERS 


Consistency 
Film Arithmetic Percent of Group Within 
No. Mean Rating 25% = 74% t10% 720% 
Z 116 43 43 57 100 
a 123 57 71 71 100 
5 119 14 57 86 100 
“ 1l2 43 57 71 100 
5 126 43 57 71 86 
6 141 14 14 71 86 
7 104 435 43 57 100 
8 107 71 86 100 100 
3 102 29 57 71 86 
10 120 86 86 100 100 
ll 121 71 71 71 100 
12 139 71 71 71 100 
13 121 43 71 100 100 
14 117 57 100 100 100 
1§ 120 71 71 100 100 
16 120 71 71 71 100 
17 124 71 86 100 100 
18 123 71 71 71 fee 
Average 5328 65.7 80.0 7 
Accuracy 
Film Corrected Percent of Group Within 
No. Mean Rating 25% 74% =10% =20 
1 112 29 57 57 86 
2 113 14 Ze 67 86 
3 131 $7 67 57 86 
4 95 14 14 29 43 
5 119 14 14 57 100 
6 157 57 57 71 100 
7 91 14 2° 29 71 
8 102 43 86 86 100 
9 117 0 14 29 86 
10 109 14 43 43 100 
11 117 43 71 71 86 
12 150 43 57 86 86 
13 113 43 43 57 100 
14 125 43 57 71 100 
15 119 71 86 100 100 
16 114 57 §7 86 100 
17 122 14 86 100 100 
18 122 71 7 71 100 


Average 35.6 51.6 64.3 90.6 
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Table 23 


SIZE OF CITY 
MORE THAN 100,000 - 11 RATERS 








Film Arithmetic Percent of Group Within 
No. Mean Rating ies 2782 2208 520% 
106 4. 4 5 hs) 
2 109 36 45 73 75 
3 100 36 45 45 73 
4 04 27 36 55 B2 
6 112 27 55 55 Be 
6 127 45 45 64 9) 
7 104 9 36 36 82 
8 106 27 36 356 82 
4 Q° 9 Q 18 B82 
10 109 18 27 64 82 
11 wl4 9 45 45 vf) 
12 135 36 64 73 73 
13 107 27 456 55 73 
14 103 46 45 55 73 
15 111 36 55 64 Be 
16 105 45 §5 64 73 
17 108 3 18 55 82 
18 113 27 64 Wi) 82 
Average 28.5 42.8 54.7 78.5 
Accuracy 
Film Corrected Percent of Group Within 
No. Mean Rating +54 274% 210% ~20% 
1 98 27 64 73 73 
i 100 55 64 64 73 
3 115 e) 6) ¢) 45 
4 84 45 55 73 82 
5 105 65 64 73 73 
6 138 18 36 36 62 
7 89 46 45 64 73 
8 100 18 45 64 82 
°] 115 18 27 27 45 
10 L103 55 55 73 82 
11 111 27 27 45 73 
12 142 27 27 36 73 
13 L102 45 45 73 73 
14 112 0 18 36 55 
15 107 45 55 64 82 
16 101 55 64 64 73 
17 109 9 27 27 82 
18 109 64 13 73 82 


Ra 


Average 33.8 44.0 53.6 72.4 
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Experience 

Concept of 
Standard 

Initial Training 
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Area 

Size of City 
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Table 24 
RESULTS OF F TEST 


Consistency 
Percent of Group Within 
25% +742 *10% 


F Signif. F Signif. F Signif. 
Value Level Value Level Value Level 


2267 - ¥ 1.93 — 


= 2.78 = 4.20 ? 
4.89 5% 16.25 1% =—s:«11.28 1g 


S215 = 2-158 e = 
52435 1% 5069 5% 7.05 1% 
6.82 1¢ 7.82 1% 7.60 1% 


Accuracy 
Percent of Group Within 
+54 = 78S 210% 


PF Signif. F Signif. F Signif. 
Value Level Value Level Value Level 


>= pa 1.89 > 
4.04 * 334 . 2.98 - 
3.16 < 6.13 5% 10.665 1% 
1.45 “ - 2.06 = 

@ e S25 = 

. 1.05 . 2e1l 7 


EXPERIENCE, CONCEPT OF STANDARD, NUMBER OF EMPLOYEES have 
no significant differences between means at the 5 percent 
confidence level in either CONSISTENCY or ACCURACY. 


TRAINING, only, has a@ significant difference between means 
at the & percent confidence level in ACCURACY. 
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Table 25 


RESULTS OF FISHER t TEST 


Confidence Level Between Pairs Of Menas 


Consistency Only 


Initial Training 
Company Vs. College” 


Area 


Michigan vs. Northern Midwest 
Michigan vs. Central Midwest 
Michigen vs. Southern Midwest 


All other pairs 


Size of City 


Over 100,000 va. 0-5,000 
Over 100,000 va. 510,000 
Over 100,000 vs. 10-25,000 
Over 100,000 vs. 25-650,000 
Over 100,000 vs. 60-100,000 


5-10,000 vs. 10«25,000 
5-10,000 vs. 25-50,000 
6-10,000 ve. 50-100,000 
0-5,000 vs. 25-60,000 


All other pairs 


Percent of Group Within 


=5% t 74S 210% 
5% 1é 14 
1% 1% 
1% . 1% 
1% - 1% 
; | 8 
1% 1% 1% 
1% 1% 1% 
. 5% 5% 
« 
ik is Y 
5¢ 1¢ - 


"significant difference between means at the 1 percent 
confidence level for ACCURACY also. 
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